Writing the Pinata CLI in Go

If you’re a developer then there’s a strong chance you do a fair bit of work in the terminal. We use it for things like making new directories or files, cloning Git repositories, or setting up a new app with a command like npx create-pinata-app (try that one by the way). To help streamline work in the terminal many people also utilize Command Line Interfaces or CLI’s. They work like programs or shortcuts to multiple different script files or binaries to accomplish an array of different tasks. Pinata has had a Node.js CLI for several years that allowed people to upload files to IPFS. It’s been well loved, however not without its problems.

One of the downsides to the older Pinata CLI is that it was written in Node.js. Now don’t get me wrong, JavaScript has some pretty incredible flexibility to do lots of things, but it sometimes comes at the cost of performance. It was this notion that spurred us to rewrite the CLI in Golang (aka Go), a language known for its speed and efficiency. It is actually the language most IPFS nodes are implemented with. Today, we’re sharing “the what” went into this rewrite so anyone can implement Go-based dev tools today.

Starting Small

When embarking on a project like this, the best thing you can do is start small. Focus on the smallest step and then slowly grow into more complex goals. Since we already had an existing CLI, we knew what functionality it would have to have. First, we needed to be able to authenticate a user with the Pinata JWT. To start we just wrote a simple Go program to make sure we could run a function. Then, we made a quick little function to test authentication with an API request.

package main

import (
	"fmt"
	"log"
	"net/http"
)

func TestAuthentication() {

	req, err := http.NewRequest("GET", "https://api.pinata.cloud/data/testAuthentication", nil)
	if err != nil {
		log.Fatal("err with request", err)
	}

	req.Header.Set("Authorization", "Bearer "+JWT)

	client := &http.Client{}
	resp, err := client.Do(req)
	if err != nil {
		log.Fatal("error sending request", err)
	}

	defer resp.Body.Close()
status := resp.StatusCode
	if status == 200 {
		fmt.Println("testAuthentication: ✅")
	} else {
		fmt.Println("testAuthentication: ❌", status)
	}
}

Now that we had this and could run it with the JWT variable, we needed to adjust it to accept a JWT, write it to the user’s dot files, then read it from that file and make sure it worked the same way.

package main

import (
	"errors"
	"fmt"
	"net/http"
	"os"
	"path/filepath"
	"time"
)

func SaveJWT(jwt string) error {
	home, err := os.UserHomeDir()
	if err != nil {
		return err
	}
	p := filepath.Join(home, ".pinata-go-cli")
	err = os.WriteFile(p, []byte(jwt), 0600)
	if err != nil {
		return err
	}
	host := GetHost()
	url := fmt.Sprintf("https://%s/data/testAuthentication", host)
	req, err := http.NewRequest("GET", url, nil)
	if err != nil {
		return err
	}

	req.Header.Set("Authorization", "Bearer "+jwt)

	client := &http.Client{
		Timeout: time.Duration(time.Second * 3),
	}
	resp, err := client.Do(req)
	if err != nil {
		return err
	}

	defer resp.Body.Close()
	status := resp.StatusCode
	if status != 200 {
		return errors.New("Authentication failed, make sure you are using the Pinata JWT")
	}

	fmt.Println("Authentication Successful!")
	return nil
}

func GetHost() string {
	return GetEnv("PINATA_HOST", "api.pinata.cloud")
}

func GetEnv(key, defaultValue string) string {
	value := os.Getenv(key)
	if len(value) == 0 {
		return defaultValue
	}
	return value
}

With the JWT saved to the user’s computer, we can now just read it for all of our other CLI functions rather than making them provide it every single time.

Adding Complexity

Now that we have authorization taken care of it was time to try something a bit harder: uploading a file. While it can sound a little daunting, all it takes is a bit of research and perhaps AI to figure out the best way to send a file via an API request. Then we just have to conform it to what the Pinata API expects. Since we wanted this upload function to accept files or folders without distinction, we made a helper function to check what the input was and append it accordingly.

func pathsFinder(filePath string, stats os.FileInfo) ([]string, error) {
	var err error
	files := make([]string, 0)
	fileIsASingleFile := !stats.IsDir()
	if fileIsASingleFile {
		files = append(files, filePath)
		return files, err
	}
	err = filepath.Walk(filePath,
		func(path string, info os.FileInfo, err error) error {
			if err != nil {
				return err
			}
			if !info.IsDir() {
				files = append(files, path)
			}
			return nil
		})

	if err != nil {
		return nil, err
	}

	return files, err
}

We also made a separate function to create the multipart request you normally need for file uploads. In it, we take in multiple arguments for the files, the version of the CID, and the name you would like for the file to be called once uploaded to Pinata.

func createMultipartRequest(filePath string, files []string, body io.Writer, stats os.FileInfo, version int, name string) (string, error) {
	contentType := ""
	writer := multipart.NewWriter(body)

	fileIsASingleFile := !stats.IsDir()
	for _, f := range files {
		file, err := os.Open(f)
		if err != nil {
			return contentType, err
		}
		defer func(file *os.File) {
			err := file.Close()
			if err != nil {
				log.Fatal("could not close file")
			}
		}(file)

		var part io.Writer
		if fileIsASingleFile {
			part, err = writer.CreateFormFile("file", filepath.Base(f))
		} else {
			relPath, _ := filepath.Rel(filePath, f)
			part, err = writer.CreateFormFile("file", filepath.Join(stats.Name(), relPath))
		}
		if err != nil {
			return contentType, err
		}
		_, err = io.Copy(part, file)
		if err != nil {
			return contentType, err
		}
	}

	pinataOptions := Options{
		CidVersion: version,
	}

	optionsBytes, err := json.Marshal(pinataOptions)
	if err != nil {
		return contentType, err
	}
	err = writer.WriteField("pinataOptions", string(optionsBytes))

	if err != nil {
		return contentType, err
	}

	pinataMetadata := Metadata{
		Name: func() string {
			if name != "nil" {
				return name
			}
			return stats.Name()
		}(),
	}
	metadataBytes, err := json.Marshal(pinataMetadata)
	if err != nil {
		return contentType, err
	}
	_ = writer.WriteField("pinataMetadata", string(metadataBytes))
	err = writer.Close()
	if err != nil {
		return contentType, err
	}

	contentType = writer.FormDataContentType()

	return contentType, nil
}

Finally, we can put it all together for an Upload function that handles errors, formats the response, and prints results to the user.

func Upload(filePath string, version int, name string, cidOnly bool) (UploadResponse, error) {
	jwt, err := findToken()
	if err != nil {
		return UploadResponse{}, err
	}

	stats, err := os.Stat(filePath)
	if os.IsNotExist(err) {
		fmt.Println("File or folder does not exist")
		return UploadResponse{}, errors.Join(err, errors.New("file or folder does not exist"))
	}

	files, err := pathsFinder(filePath, stats)
	if err != nil {
		return UploadResponse{}, err
	}

	body := &bytes.Buffer{}
	contentType, err := createMultipartRequest(filePath, files, body, stats, version, name)
	if err != nil {
		return UploadResponse{}, err
	}

	totalSize := int64(body.Len())
	fmt.Printf("Uploading %s (%s)\n", stats.Name(), formatSize(int(totalSize)))

	progressBody := newProgressReader(body, totalSize)

	host := GetHost()
	url := fmt.Sprintf("https://%s/pinning/pinFileToIPFS", host)
	req, err := http.NewRequest("POST", url, progressBody)
	if err != nil {
		return UploadResponse{}, errors.Join(err, errors.New("failed to create the request"))
	}
	req.Header.Set("Authorization", "Bearer "+string(jwt))
	req.Header.Set("content-type", contentType)

	client := &http.Client{}
	resp, err := client.Do(req)
	if err != nil {
		return UploadResponse{}, errors.Join(err, errors.New("failed to send the request"))
	}
	if resp.StatusCode != 200 {
		return UploadResponse{}, fmt.Errorf("server Returned an error %d", resp.StatusCode)
	}
	err = progressBody.bar.Set(int(totalSize))
	if err != nil {
		return UploadResponse{}, err
	}
	fmt.Println()
	defer func(Body io.ReadCloser) {
		err := Body.Close()
		if err != nil {
			log.Fatal("could not close request body")
		}
	}(resp.Body)

	var response UploadResponse
	err = json.NewDecoder(resp.Body).Decode(&response)
	if err != nil {
		return UploadResponse{}, err
	}
	if cidOnly {
		fmt.Println(response.IpfsHash)
	} else {
		fmt.Println("Success!")
		fmt.Println("CID:", response.IpfsHash)
		fmt.Println("Size:", formatSize(response.PinSize))
		fmt.Println("Date:", response.Timestamp)
		if response.IsDuplicate {
			fmt.Println("Already Pinned: true")
		}
	}
	return response, nil
}

Now that we have our main functions, the next goal is to build the CLI. Thankfully there are lots of great frameworks and tools to make the process easier, and personally, we fell in love with urfave/cli. It makes everything pretty straightforward when it comes to building commands. Here’s what our CLI would look like with just authentications and uploads.

package main

import (
	"errors"
	"github.com/urfave/cli/v2"
	"log"
	"os"
)

type UploadResponse struct {
	IpfsHash    string `json:"IpfsHash"`
	PinSize     int    `json:"PinSize"`
	Timestamp   string `json:"Timestamp"`
	IsDuplicate bool   `json:"isDuplicate"`
}

type Options struct {
	CidVersion int `json:"cidVersion"`
}

type Metadata struct {
	Name      string                 `json:"name"`
	KeyValues map[string]interface{} `json:"keyvalues"`
}

func main() {
	app := &cli.App{
		Name:  "pinata",
		Usage: "A CLI for uploading files to Pinata! To get started make an API key at https://app.pinata.cloud/keys, then authorize the CLI with the auth command with your JWT",
		Commands: []*cli.Command{
			{
				Name:      "auth",
				Aliases:   []string{"a"},
				Usage:     "Authorize the CLI with your Pinata JWT",
				ArgsUsage: "[your Pinata JWT]",
				Action: func(ctx *cli.Context) error {
					jwt := ctx.Args().First()
					if jwt == "" {
						return errors.New("no jwt supplied")
					}
					err := SaveJWT(jwt)
					return err
				},
			},
			{
				Name:      "upload",
				Aliases:   []string{"u"},
				Usage:     "Upload a file or folder to Pinata",
				ArgsUsage: "[path to file]",
				Flags: []cli.Flag{
					&cli.IntFlag{
						Name:    "version",
						Aliases: []string{"v"},
						Value:   1,
						Usage:   "Set desired CID version to either 0 or 1. Default is 1.",
					},
					&cli.StringFlag{
						Name:    "name",
						Aliases: []string{"n"},
						Value:   "nil",
						Usage:   "Add a name for the file you are uploading. By default it will use the filename on your system.",
					},
          &cli.BoolFlag{
            Name: "cid-only",
            Usage: "Use if you only want the CID returned after an upload",
          },
				},
				Action: func(ctx *cli.Context) error {
					filePath := ctx.Args().First()
					version := ctx.Int("version")
					name := ctx.String("name")
          cidOnly := ctx.Bool("cid-only")
					if filePath == "" {
						return errors.New("no file path provided")
					}
					_, err := Upload(filePath, version, name, cidOnly)
					return err
				},
			},
		},
	}

	if err := app.Run(os.Args); err != nil {
		log.Fatal(err)
	}
}

The urfave framework allows you to set up different commands with names, help tips, and more. You can also include flags for certain logic, so for instance, on upload, you can run pinata upload --version 0 and it would return a V0 CID QmVLwvmGehsrNEvhcCnnsw5RQNseohgEkFNN1848zNzdng vs the default V1 CID bafkreih5aznjvttude6c3wbvqeebb6rlx5wkbzyppv7garjiubll2ceym4. With this, we now have our MVP!

Expanding Further

Now that we have a general pattern and structure for our CLI, adding other commands is easy! For instance, to add a delete command to delete a file, we just add a new file called Delete.go and add in our code to make a request with the CID as the argument.

package main

import (
	"errors"
	"fmt"
	"net/http"
)

func Delete(cid string) error {
	jwt, err := findToken()
	if err != nil {
		return err
	}
	host := GetHost()
	url := fmt.Sprintf("https://%s/pinning/unpin/%s", host, cid)

	req, err := http.NewRequest("DELETE", url, nil)
	if err != nil {
		return errors.Join(err, errors.New("failed to create the request"))
	}
	req.Header.Set("Authorization", "Bearer "+string(jwt))
	req.Header.Set("content-type", "application/json")

	client := &http.Client{}
	resp, err := client.Do(req)
	if err != nil {
		return errors.Join(err, errors.New("failed to send the request"))
	}
	defer resp.Body.Close()

	if resp.StatusCode != 200 {
		return fmt.Errorf("server Returned an error %d, check CID", resp.StatusCode)
	}

	fmt.Println("File Deleted")

	return nil

}

Then we just add that command to our main list of commands for the CLI.

{
	Name:      "delete",
	Aliases:   []string{"d"},
	Usage:     "Delete a file by CID",
	ArgsUsage: "[CID of file]",
	Action: func(ctx *cli.Context) error {
		cid := ctx.Args().First()
		if cid == "" {
			return errors.New("no CID provided")
		}
		err := Delete(cid)
		return err
	},
},

With this, we can run pinata delete QmVLwvmGehsrNEvhcCnnsw5RQNseohgEkFNN1848zNzdng and it will unpin the file from our account. But what if you don’t have the CID? Well, let’s add in a file list too!

package main

import (
	"encoding/json"
	"errors"
	"fmt"
	"net/http"
)

func ListFiles(amount string, cid string, name string, status string, offset string) (ListResponse, error) {
	jwt, err := findToken()
	if err != nil {
		return ListResponse{}, err
	}
	host := GetHost()
	url := fmt.Sprintf("https://%s/data/pinList?includesCount=false&pageLimit=%s&status=%s", host, amount, status)

	if cid != "null" {
		url += "&hashContains=" + cid
	}
	if name != "null" {
		url += "&metadata[name]=" + name
	}
	if offset != "null" {
		url += "&pageOffset=" + offset
	}

	req, err := http.NewRequest("GET", url, nil)
	if err != nil {
		return ListResponse{}, errors.Join(err, errors.New("failed to create the request"))
	}
	req.Header.Set("Authorization", "Bearer "+string(jwt))
	req.Header.Set("content-type", "application/json")

	client := &http.Client{}
	resp, err := client.Do(req)
	if err != nil {
		return ListResponse{}, errors.Join(err, errors.New("failed to send the request"))
	}
	defer resp.Body.Close()

	if resp.StatusCode != 200 {
		return ListResponse{}, fmt.Errorf("server Returned an error %d", resp.StatusCode)
	}

	var response ListResponse

	err = json.NewDecoder(resp.Body).Decode(&response)
	if err != nil {
		return ListResponse{}, err
	}
	formattedJSON, err := json.MarshalIndent(response.Rows, "", "    ")
	if err != nil {
		return ListResponse{}, errors.New("failed to format JSON")
	}

	fmt.Println(string(formattedJSON))

	return response, nil

}

This one is interesting as the /data/pinList endpoint has a lot of acceptable queries, so depending on what the user enters we’ll add in several here and add the command to our CLI like the others.

{
	Name:    "list",
	Aliases: []string{"l"},
	Usage:   "List most recent files",
	Flags: []cli.Flag{
		&cli.StringFlag{
			Name:    "cid",
			Aliases: []string{"c"},
			Value:   "null",
			Usage:   "Search files by CID",
		},
		&cli.StringFlag{
			Name:    "amount",
			Aliases: []string{"a"},
			Value:   "10",
			Usage:   "The number of files you would like to return, default 10 max 1000",
		},
		&cli.StringFlag{
			Name:    "name",
			Aliases: []string{"n"},
			Value:   "null",
			Usage:   "The name of the file",
		},
		&cli.StringFlag{
			Name:    "status",
			Aliases: []string{"s"},
			Value:   "pinned",
			Usage:   "Status of the file. Options are 'pinned', 'unpinned', or 'all'. Default: 'pinned'",
		},
		&cli.StringFlag{
			Name:    "pageOffset",
			Aliases: []string{"p"},
			Value:   "null",
			Usage:   "Allows you to paginate through files. If your file amount is 10, then you could set the pageOffset to '10' to see the next 10 files.",
		},
	},
	Action: func(ctx *cli.Context) error {
		cid := ctx.String("cid")
		amount := ctx.String("amount")
		name := ctx.String("name")
		status := ctx.String("status")
		offset := ctx.String("pageOffset")
		_, err := ListFiles(amount, cid, name, status, offset)
		return err
	},
},

With this command, I can easily fetch my 10 latest files with pinata list! We purposely make the output of the command to be formatted JSON so developers can pipe it into files or other services, as well as plenty of flags to search and filter their results.

Wrapping Up

After we had built out this CLI and tested it we noticed quite a boost in performance compared to its Node.js predecessor, and it gave us the opportunity to explore what else we could make it do. The result is a CLI that mimics most of the Pinata API and gives you the tools you need to extend your IPFS workflow. It’s been so fun and easy to use that I find myself using it in my own workflows all the time for projects like Cosmic Cowboys and beyond. Don’t take my word for it; try it out now! Just visit this link and follow the download instructions and usage guide to get started.

This project is just the beginning as we plan to port the Pinata API into a new Typescript SDK, as well as other programming languages. Our community has done a great job making their own implementations, but we’re excited to focus on developers and continue supporting them no matter what language they’re working in.

Happy Pinning!