- 488 Views
- 1 replies
- 1 kudos
SQL to Spark Dataframe
Hi there,Have a simple question. Not sure if Databricks supports this, but I'm wondering if there's a way to store the results of a sql cell into a spark dataframe? Or vice-versa, is there a way to take a sql query in python (saved as a string variab...
- 488 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @ChristianRRL, results of sql cell are automatically made available as a python dataframe using the _sqldf variable. You can read more about it here. For the second part not sure why you would need it when you can simply run the query like:spark.s...
- 1 kudos
- 3680 Views
- 2 replies
- 1 kudos
Catalog issue
When i was trying to create catalog i got an error saying to mention azure storage account and storage container in the following query -CREATE CATALOG IF NOT EXISTS Databricks_Anu_Jal_27022024MANAGED LOCATION 'abfss://<databricks-workspace-stack-anu...
- 3680 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @anushajalesh28, To create a catalog in Azure Databricks, you need to specify the Azure storage account and storage container in the MANAGED LOCATION clause. Let’s break down the query: CREATE CATALOG IF NOT EXISTS Databricks_Anu_Jal_270220...
- 1 kudos
- 8050 Views
- 6 replies
- 0 kudos
Resolved! Clusters are failing to launch
Hi Guys,I am a complete newbie to data bricks, we are trying to figure out if our data models and ETL can run on it.I have got the failure to launch message. I have read this message as well.https://community.databricks.com/t5/data-engineering/cluste...
- 8050 Views
- 6 replies
- 0 kudos
- 0 kudos
Hi @Peter_Jones, Let’s tackle this cluster launch issue step by step. Quota Exceeded Error: The error message indicates that your cluster launch failed due to exceeding the approved quota for standardEDSv4Family Cores in the westeurope location. ...
- 0 kudos
- 3059 Views
- 2 replies
- 0 kudos
Resolved! spark context in databricks
Hi @all,In Azure Databricks,I am using structured streaming for each batch functionality, in one of the functions I am creating tempview with pyspark dataframe (*Not GlobalTempView) and trying to access the same temp view by using spark.sql functiona...
- 3059 Views
- 2 replies
- 0 kudos
- 0 kudos
Do you face this issue without spark streaming as well? Also, could you share a minimal repo code preferably without streaming?
- 0 kudos