본문 바로가기

전체 글

 (33)
linux command file permission check $ls -l filename r : read permission. w : write permission. x : executable permission. First combination of rwx represents permission for the owner . Second combination of rwx represents permission for the group . Third combination of rwx represents permission for the other of the file. Octal notation (8진법 숫자로 표현) Read or r is represented by 4 Write or w is represented by 2 Execute x is represent..
[json] pandas dataframe to json file / json file read, write, append 1. pandas dataframe -> json file df.to_json("filename.json", default_handler=str) df.to_json("filename.json", oriend='records', default_handler=str) 2. json file -> pandas dataframe df = pd.read_json('filename.json') ** large file ** import sys maxparse = sys.maxsize while True: try: df = pd.read_json("filename.json", dtype=object) break except OverflowError: maxparse = int(maxparse/10) 3. json ..
[pyspark] create spark dataframe from pyspark.sql.types import * schema = StructType([StructField("col1",StringType(),True),\ StructField("col2",IntegerType(),True),\ StructField("col3",DoubleType(),True),\ StructField("col4",FloatType(),True)]) rowdata = [] rowdata.append(("col1_value",222,1.23,3.44555)) ... createDf = spark.createDataFrame(rowdata,schema)

티스토리툴바