- 11800 Views
- 1 replies
- 0 kudos
TypeError: Column is not iterable when using more than one columns in withColumn()
I am trying to find quarter start date from a date column. I get the expected result when i write it using selectExpr() but when i add the same logic in .withColumn() i get TypeError: Column is not iterableI am using a workaround as follows workarou...
- 11800 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi, This is a super old question but answering in case anyone else comes across it. This isn't working because the add months expects an integer rather than a column name you can get round this by using expr() inside withColumn: from pyspark.sql.func...
- 0 kudos
- 13121 Views
- 2 replies
- 0 kudos
- 13121 Views
- 2 replies
- 0 kudos
- 0 kudos
These are action methods that return data -first() : Returns the very first row of the dataframe as a single row.head() : This does the same as first(), returns the first rowhead(n): Returns an array or list of the first n rowstake(n): Similar to hea...
- 0 kudos
- 29955 Views
- 3 replies
- 0 kudos
spark is case sensitive? Spark is not case sensitive by default. If you have same column name in different case (Name, name), if you try to select eit...
spark is case sensitive?Spark is not case sensitive by default. If you have same column name in different case (Name, name), if you try to select either "Name" or "name" column you will get column ambiguity error.There is a way to handle this issue b...
- 29955 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi I had similar issues with parquet files when trying to query athena, fix was i had to inspect the parquet file since it contained columns such as "Name", "name" which the aws crawler / athena would interpret as a duplicate column since it would se...
- 0 kudos
- 9908 Views
- 5 replies
- 5 kudos
Support of running multiple cells at a time in databricks notebook Hi all,Now databricks notebook supports parallel run of commands in a single notebo...
Support of running multiple cells at a time in databricks notebookHi all,Now databricks notebook supports parallel run of commands in a single notebook that will help run ad hoc queries simultaneously without creating a separate notebook.Once you run...
- 9908 Views
- 5 replies
- 5 kudos
- 5 kudos
Hi Team,I am observing that the functionality is not working as expected in the Trial workspace of Databricks. Is there a setting that needs to be enabled to allow independent SQL cells in a Databricks notebook to run in parallel, while dependent cel...
- 5 kudos
- 35353 Views
- 7 replies
- 13 kudos
How to convert string to datetime with correct timezone?
I have a field stored as a string in the format "12/30/2022 10:30:00 AM"If I use the function TO_DATE, I only get the date part... I want the full date and time.If I use the function TO_TIMESTAMP, I get the date and time, but it's assumed to be UTC, ...
- 35353 Views
- 7 replies
- 13 kudos
- 13 kudos
use from_utc_timestamp(to_timestam("<string>", <format>),<timezone>)
- 13 kudos
- 9702 Views
- 7 replies
- 4 kudos
need help with Azure Databricks questions on CTE and SQL syntax within notebooks
Hi amazing community folks,Feel free to share your experience or knowledge regarding below questions:-1.) Can we pass a CTE sql statement into spark jdbc? i tried to do it i couldn't but i can pass normal sql (Select * from ) and it works. i heard th...
- 9702 Views
- 7 replies
- 4 kudos
- 4 kudos
Not a comparison, but there is a DB-SQL cheatsheet at https://www.databricks.com/sites/default/files/2023-09/databricks-sql-cheatsheet.pdf/
- 4 kudos
- 5322 Views
- 4 replies
- 0 kudos
Spark last window dont flush in append mode
The problem is very simple, when you use TUMBLING window with append mode, then the window is closed only when the next message arrives (+watermark logic). In the current implementation, if you stop incoming streaming data, the last window will NEVER...
- 5322 Views
- 4 replies
- 0 kudos
- 16906 Views
- 2 replies
- 0 kudos
- 16906 Views
- 2 replies
- 0 kudos
- 0 kudos
Is the limit per "table/dataframe" or for all tables/dataframes put together?The driver collects the data from all executors (which are having the respective table or dataframe) and distributes to all executors. When will the memory be released in bo...
- 0 kudos
- 5416 Views
- 4 replies
- 4 kudos
Resolved! Database creation error
I am trying to create database with external location abfss but facing the below error.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Got exception: shaded.databricks.azurebfs.org.apache.hadoop.fs.azurebfs....
- 5416 Views
- 4 replies
- 4 kudos
- 4 kudos
Changing it to a CLUSTER level for OAuth authentication helped me solve the problem.I wish the notebook AI bot could tell me the solution.before the changes, my configraiotn was at the notebook leve.and it has below errorsAnalysisException: org.apac...
- 4 kudos
- 18822 Views
- 3 replies
- 7 kudos
Resolved! collect_list by preserving order based on another variable - Spark SQL
I am using databricks sql notebook to run these queries. I have a Python UDF like %python from pyspark.sql.functions import udf from pyspark.sql.types import StringType, DoubleType, DateType def get_sell_price(sale_prices): return sale_...
- 18822 Views
- 3 replies
- 7 kudos
- 7 kudos
I had a similar situation where I was trying to order the days of the week from Monday to Sunday. I saw solutions that use Python but was wanting to do it all in SQL. My original attempt was to use: CONCAT_WS(',', COLLECT_LIST(DISTINCT t.LOAD_ORIG_...
- 7 kudos
- 17814 Views
- 8 replies
- 12 kudos
Resolved! I cannot see the Hive databases or tables once I terminate the cluster and use another cluster.
I am using Databricks community edition for learning purposes. I created some Hive-managed tables through spark sql as well as with df.saveAsTable options. But when I connect to a new cluser, "Show databases" only returns the default database....
- 17814 Views
- 8 replies
- 12 kudos
- 12 kudos
As the file still in the dbfs you can just recreate the reference of your tables and continue the work, with something like this:db_name = "mydb" from pathlib import Path path_db = f"dbfs:/user/hive/warehouse/{db_name}.db/" tables_dirs = dbutils.fs.l...
- 12 kudos
- 17229 Views
- 2 replies
- 6 kudos
Resolved! CREATE TEMP TABLE FROM CTE
I have written a CTE in Spark SQL WITH temp_data AS ( ...... ) CREATE VIEW AS temp_view FROM SELECT * FROM temp_view; I get a cryptic error. Is there a way to create a temp view from CTE using Spark SQL in databricks?
- 17229 Views
- 2 replies
- 6 kudos
- 6 kudos
In the CTE you can't do a CREATE. It expects an expression in the form of expression_name [ ( column_name [ , ... ] ) ] [ AS ] ( query )where expression_name specifies a name for the common table expression.If you want to create a view from a CTE, y...
- 6 kudos
- 8481 Views
- 4 replies
- 0 kudos
Unable to save file in DBFS
I have took the azure datasets that are available for practice. I got the 10 days data from that dataset and now I want to save this data into DBFS in csv format. I have facing an error :" No such file or directory: 'No such file or directory: '/dbfs...
- 8481 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi,after some exercise you need to aware folder create in dbutils.fs.mkdirs("/dbfs/tmp/myfolder") it's created in /dbfs/dbfs/tmp/myfolderif you want to access path to_csv("/dbfs/tmp/myfolder/mytest.csv") you should created with this script dbutils.fs...
- 0 kudos
- 20392 Views
- 5 replies
- 7 kudos
Resolved! Window function using last/last_value with PARTITION BY/ORDER BY has unexpected results
Hi, I'm wondering if this is the expected behavior when using last or last_value in a window function? I've written a query like this:select col1, col2, last_value(col2) over (partition by col1 order by col2) as column2_last from values ...
- 20392 Views
- 5 replies
- 7 kudos
- 7 kudos
For those stumbling across this; it seems LAST_VALUE emulates the same functionality as it does in SQL Server which does not, in most people's minds, have a proper row/range frame for the window. You can adjust it with the below syntax.I understand l...
- 7 kudos
- 4388 Views
- 4 replies
- 1 kudos
sample
Help parsing the JSON using Spark SQL or python. Sample json attached.
- 4388 Views
- 4 replies
- 1 kudos