Golang创建可能损坏的文件
I use the following code which writes YAML content to a file, we have an old tool in java that do exactly the same and now we are building new tool which will do the same in Golang.
To create a file with the content I use
err := ioutil.WriteFile(tmpDirPath, fileContent, 0777)
I don't get any error from this process
I was able to see the file content and it looks OK (exactly like the file that was created in the old tool) We have an internal tool which failed to read the file. The error is:
Error while retrieving archive entry
I check the content (with file diff) and I see that it exactly the same (from the java and Go tools), I'm really Frustrated :( because I don't know how to proceed, maybe the file is corrupted. I've also tried with file.Mode = 0644 and it doesn't help.
I've Mac and if I can run some checks with command line or other tool to see the diffrent between the files or maybe between the zip's, (which is created by the old and the new tool) please let me know what to do ...
Btw, after I've created the files, I'm, zipping the files and provide it to the internal tool to use it.
If the zip code it needs and can help to identify the problem I can share it also if needed
To create the dir that the file is inside I use
func CreateDirIfNotExist(dir string) {
if _, err := os.Stat(dir); os.IsNotExist(err) {
err := os.MkdirAll(dir, 0777)
if err != nil {
panic(err)
}
}
}
To zip all content I use after I finish processing:
func Zipit(params ...string) error {
zipfile, err := os.Create(params[1])
if err != nil {
return err
}
defer zipfile.Close()
archive := zip.NewWriter(zipfile)
defer archive.Close()
info, err := os.Stat(params[0])
if err != nil {
return nil
}
var baseDir string
if info.IsDir(); len(params) > 2 {
baseDir = params[2]
} else {
baseDir = filepath.Base(params[0])
}
filepath.Walk(params[0], func(path string, info os.FileInfo, err error) error {
if err != nil {
return err
}
header, err := zip.FileInfoHeader(info)
if err != nil {
return err
}
if baseDir != "" {
header.Name = filepath.Join(strings.TrimPrefix(path, params[0]))
}
if info.IsDir() {
header.Name += "/"
} else {
header.Method = zip.Deflate
}
writer, err := archive.CreateHeader(header)
if err != nil {
return err
}
if info.IsDir() {
return nil
}
file, err := os.Open(path)
if err != nil {
return err
}
defer file.Close()
_, err = io.Copy(writer, file)
return err
})
return err
}
update
I've this folder in my MAC
I need to build it on go that I will able to uzip it on mac without errors, I've tried two tools and I get different errors.
I use the code above to zip the content of it, I know that in the command line there is no error when unzip but our tool cannot read it... what am I doing wrong with the zip code ? (I need it without the base dir compression )
some errors is: with the mac default archive tool and other tool
[![enter image description here][2]][2]
UPDATE 2
As requested this is my whole program which zip the folder successfully after the process done but with the MAC tools it's not possible to unzip them, see the errors above with two diffrent utils, and this dont happen (with mac tools) for other zip files...
package main
import (
"archive/zip"
"fmt"
"io"
"log"
"os"
"os/exec"
"path/filepath"
"strings"
"bufio"
"unicode/utf8"
)
func Zipit(params ...string) error {
zipfile, err := os.Create(params[1])
if err != nil {
return err
}
defer zipfile.Close()
archive := zip.NewWriter(zipfile)
defer archive.Close()
info, err := os.Stat(params[0])
if err != nil {
return err
}
var baseDir string
if info.IsDir(); len(params) > 2 {
baseDir = params[2]
} else {
baseDir = filepath.Base(params[0])
}
filepath.Walk(params[0], func(path string, info os.FileInfo, err error) error {
if err != nil {
return err
}
header, err := zip.FileInfoHeader(info)
if err != nil {
return err
}
fmt.Printf("header: %#v
", header)
if baseDir != "" {
header.Name = filepath.Join(strings.TrimPrefix(path, params[0]))
}
if info.IsDir() {
header.Name += "/"
} else {
header.Method = zip.Deflate
}
writer, err := archive.CreateHeader(header)
if err != nil {
return err
}
if info.IsDir() {
return nil
}
file, err := os.Open(path)
if err != nil {
return err
}
defer file.Close()
_, err = io.Copy(writer, file)
return err
})
return err
}
func CreateDirIfNotExist(dir string) {
if _, err := os.Stat(dir); os.IsNotExist(err) {
err := os.MkdirAll(dir, 0777)
if err != nil {
panic(err)
}
}
}
func main() {
archiveName := "ui5.pre.zip"
fixedArchiveName := "ui5.zip"
cmdParams := [][]string{
{"./ui5", "npm", "install"},
}
for _, cp := range cmdParams {
log.Printf("[INFO] - Starting %s in folder %s...", cp[1:], cp[0])
cmd := exec.Command(cp[1], cp[2:]...)
cmd.Dir = cp[0]
// Wait to finish, get output:
stdout, err := cmd.StdoutPipe()
if err != nil {
log.Panic("%s cmd.StdoutPipe() error: %v
", cp[1:], err)
}
// Start command:
cmd.Start()
// Stream the execution of the current command
scanner := bufio.NewScanner(stdout)
scanner.Split(bufio.ScanRunes)
oneRune := make([]byte, utf8.UTFMax)
for {
count, err := stdout.Read(oneRune)
if err != nil {
break
}
fmt.Printf("%s", oneRune[:count])
}
log.Println()
// Get execution success or failure:
if err := cmd.Wait(); err != nil {
log.Panic("Error running %s: %v
", cp[1:], err)
}
log.Println("Finished %s", cp[1:])
}
if err := Zipit("ui5", "ui5.pre.zip"); err != nil {
fmt.Println(err)
}
if err := exec.Command("zip", "-F", archiveName, "--out", "./"+fixedArchiveName).Run(); err != nil {
fmt.Println(err)
}
}
The problem is the same discussed https://groups.google.com/forum/#!topic/golang-nuts/0iae5Ng-I-0 and depends on ZIP64 format, which used Go archive/go.
An answer recommended using a Go wrapper for Java's ZipInputStream https://github.com/30x/zipper.
Updated: First, call
go get github.com/30x/zipper
to load zipper package.
Then try to use this version of your Zipit function to check a solution:
import "github.com/30x/zipper"
......
func Zipit(params ...string) error {
return zipper.Archive(params[0], params[1], zipper.Options{
ExcludeBaseDir: false, // or true
})
}
Or try to fix archive created your Zipit function (as in the question) to make it ZipInputStream compatible. Call the zipper.Process function next your Zipit version function call in your project:
archiveName := "archive.zip" // example code, using yours
err := Zipit(tmpDirPath, archiveName)
if err!=nil {
return err
}
err := zipper.Process(archiveName, archiveName)
if err!=nil {
return err
}
Second version to fix an archive - using zip with an option -F:
archiveName := "ui5.pre.zip"
fixedArchiveName := "ui5.zip"
if err := Zipit("ui5", archiveName); err != nil {
fmt.Println(err)
}
if err := exec.Command("zip", "-F", archiveName, "--out", fixedArchiveName).Run(); err != nil {
fmt.Println(err)
}
The result of the chat conversation. The problem is placed in two parts in your code.
First part depends on file info of archive item. Path has to be as a relative, but not absolute. File info header name needs to trim '/' in prefix.
Two piece of code needs to fix.
First:
var baseDir string
if info.IsDir(); len(params) > 2 {
baseDir = params[2]
} else {
baseDir = filepath.Base(params[0])
}
// add the line:
if len(baseDir)>0 {
baseDir += "/" // add a suffix to trim it in a file name
}
Second:
if baseDir != "" {
header.Name = filepath.Join(strings.TrimPrefix(path, baseDir)) // using baseDir to trim but not params[0]
}
Second part depends on directories added to an archive.
Skip directories at all while building an archive to get the same result as your Java tools.
filepath.Walk(params[0], func(path string, info os.FileInfo, err error) error {
if err != nil {
return err
}
// move up this piece of code
if info.IsDir() {
return nil
}
header, err := zip.FileInfoHeader(info)
if err != nil {
return err
}
if baseDir != "" {
header.Name = filepath.Join(strings.TrimPrefix(path, baseDir))
}
// simplify code - just define a method
//if info.IsDir() {
// header.Name += "/"
//} else {
header.Method = zip.Deflate
//}
writer, err := archive.CreateHeader(header)
if err != nil {
return err
}
// too late - directory is already in the archive
//if info.IsDir() {
// return nil
//}
First of all it's impossible for the files to be identical and the interpreter program to give different results. You're not writing the whole history in your description.
- How can I check if the file is corrupted ?
The answer is in the internal tool. You should improve its debug message: "Error while retrieving archive entry". What error ? Debug your old tool. And it's suggesting the problem is in the zip file, not the json file.
- Should I try to use other API to check it? if yes, which ?
Yes, unzip both files by hand and diff the output.
- Any other direction ?
Read the code for your old java tool.