- 1112 Views
- 2 replies
- 0 kudos
how to assign account level groups to workspace using, Terraform
in the workspace console when I create groups it creates a source as an account, Basically, it is a account level group, But provider "databricks" { host = var.databricks_host # client_id = "" # client_secret = " account_id = ...
- 1112 Views
- 2 replies
- 0 kudos
- 0 kudos
@chethankumar wrote:in the workspace console when I create groups it creates a source as an account, Basically, it is a account level group, But provider "databricks" { host = var.databricks_host # client_id = "" # client_secre...
- 0 kudos
- 864 Views
- 2 replies
- 2 kudos
Access to For Each run ids from jobs rest API
Hello CommunityI'm using the for_each tasks in workflows, but I'm struggling to access to the job information through the job APIs.In short, using the runs api (Get a single job run | Jobs API | REST API reference | Databricks on AWS), I'm able to ac...
- 864 Views
- 2 replies
- 2 kudos
- 2 kudos
I couldn't figure out anything to systematically access for_each tasks. I believe this is a quite important missing piece (unless of course I'm missing something).Is there a way to ask to a developer or should I open a ticket through my company?
- 2 kudos
- 5711 Views
- 13 replies
- 3 kudos
Trying to use Python source file as module in databricks Notebook
Hi everyone,I’m currently working on a project in Databricks(version 13.3 LTS) and could use some help with importing external Python files as modules into my notebook. I’m aiming to organize my code better and reuse functions across different notebo...
- 5711 Views
- 13 replies
- 3 kudos
- 3 kudos
Hi @adhi_databricks ,I am out of ideas in this case. Is utils.py the correct python file, no errors found.Could you test with some simple code like below?I am starting to think there is something wrong with the file (although you mentioned it works i...
- 3 kudos
- 885 Views
- 1 replies
- 0 kudos
Geometry Data type in sql
Hi All,I am trying to create a table with Geometry datatype column in Databricks SQL.Unfortunately, I am getting not supported data type error.Is there any way I can create a table with Geometry datatype. Thanks.
- 885 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Shan_n ,Databricks doesn't have native support for geometry data type. You can look at the list of all available data types below: https://docs.databricks.com/en/sql/language-manual/sql-ref-datatypes.htmlBut there is a way to work with geospatial...
- 0 kudos
- 950 Views
- 3 replies
- 2 kudos
Resolved! ANALYZE table for stats collection
Hi all,I understand ANALYZE table for stats collection does not interfere with write & update operations on a delta table. Please confirm.I like to execute ANALYZE table command post data loads of delta tables but at times the loads could be extended...
- 950 Views
- 3 replies
- 2 kudos
- 2 kudos
@filipniziol thanks for your time in replying. your answer is satisfactory & resolves my queries.
- 2 kudos
- 2616 Views
- 1 replies
- 0 kudos
I am trying to use spark session of the compute in java Jar to run queries against tables unity cata
I am trying to use spark session of the compute in java Jar to run queries against tables unity catalog . I get the following error SparkSession spark = SparkSession.builder() .appName("Databricks Query Example") .confi...
- 2616 Views
- 1 replies
- 0 kudos
- 0 kudos
@hemprasad newjetnet aa loginwrote:I am trying to use spark session of the compute in java Jar to run queries against tables unity catalog . I get the following error SparkSession spark = SparkSession.builder() .appName("Databricks Qu...
- 0 kudos
- 1824 Views
- 1 replies
- 1 kudos
Conversion of time zone from UTC to US/Eastern
Hi All,I am new to databricks and i am writing a query to fetch the data from Databricks and encountered with an error. Please see below the query details and the error. Any help would be appreciated. Thank You in Advance ! Query: cast(TO_UTC_TIMESTA...
- 1824 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Majid ,Spark SQL doesn't support AT TIME ZONE, that's why you've got this error. To achieve similar result, you can use to_utc_timestamp or from_utc_timestamp function. Those functions support timezone parameter:https://docs.databricks.com/en/sq...
- 1 kudos
- 35365 Views
- 7 replies
- 2 kudos
Resolved! Simply writing a dataframe to a CSV file (non-partitioned)
When writing a dataframe in Pyspark to a CSV file, a folder is created and a partitioned CSV file is created. I have then rename this file in order to distribute it my end user.Is there any way I can simply write my data to a CSV file, with the name ...
- 35365 Views
- 7 replies
- 2 kudos
- 2 kudos
I know this post is a little old, but Chat GPT actually put together a very clean and straightforward solution for me (in scala): // Set the temporary output directory and the desired final file pathval tempDir = "/tmp/your_file_name"val finalOutputP...
- 2 kudos
- 2583 Views
- 5 replies
- 3 kudos
dlt Streaming Checkpoint Not Found
I am using Delta Live Tables and have my pipeline defined using the code below. My understanding is that a checkpoint is automatically set when using Delta Live Tables. I am using the Unity Catalog and Schema settings in the pipeline as the storage d...
- 2583 Views
- 5 replies
- 3 kudos
- 3 kudos
Hi @ggsmith ,If you use Delta Live Tables then checkpoints are stored under the storage location specified in the DLT settings. Each table gets a dedicated directory under storage_location/checkpoints/<dlt_table_name.
- 3 kudos
- 928 Views
- 3 replies
- 1 kudos
Connecting to Blob storage using abfss not working with serverless compute
I tried to follow the instructions found here: Connect to Azure Data Lake Storage Gen2 and Blob Storage - Azure Databricks | Microsoft LearnE.g. this code:spark.conf.set("fs.azure.account.key.<storage-account>.dfs.core.windows.net",dbutils.secrets.ge...
- 928 Views
- 3 replies
- 1 kudos
- 746 Views
- 2 replies
- 0 kudos
Running a cell with R-script keeps waiting status
So, i have a R-notebook with different cells and a '15.4 LTS ML (includes Apache Spark 3.5.0, Scala 2.12)' cluster.If i select 'run all' all cells will be run immediately and the run finishes quickly and fine. But if i would like to run the cells one...
- 746 Views
- 2 replies
- 0 kudos
- 0 kudos
Today, I tried the glm function from the SparkR package. And it seemed to have initially solved the problem with the glm function. However, when you save the result of the glm function in a variable, things seem to go wrong. But only when the variabl...
- 0 kudos
- 804 Views
- 2 replies
- 1 kudos
SQL merge to update one of the nested column
I am having existing delta-lake as target, and the small set of records at hand as CURRENT_BATCH,I have a requirement to update dateTimeUpdated column inside parent2, using following merge query.========MERGE INTO mydataset AS targetUSING CURRENT_BA...
- 804 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @SagarJi ,According to the documentation updates to the nested columns are not supported:What you can do you can construct the whole struct and update the parent:MERGE INTO mydataset AS target USING CURRENT_BATCH AS incoming ON target.parent1.comp...
- 1 kudos
- 10002 Views
- 6 replies
- 3 kudos
Resolved! SQL Warehouse Serverless - Not able to access the external tables in the hive_metastore
I have DLT tables created under the hive_metastore with external data stored in ADL gen2.The ADL blob storage is mounted into /mnt/<storage-account>The tables are successfully created and accessible from my notebooks, as well the ADL storage.I have c...
- 10002 Views
- 6 replies
- 3 kudos
- 729 Views
- 1 replies
- 1 kudos
MalformedInputException when using extended ascii characters in dbutils.notebook.exit()
I have a specific use case where I call another notebook using the dbutils.notebook.run() function. The other notebook do some processing and return a string in the dbutils.notebook.exit() function to the caller notebook. The returned string has some...
- 729 Views
- 1 replies
- 1 kudos
- 1 kudos
@jfpatenaude starbuckssecretmenu wrote:I have a specific use case where I call another notebook using the dbutils.notebook.run() function. The other notebook do some processing and return a string in the dbutils.notebook.exit() function to the caller...
- 1 kudos
- 23716 Views
- 2 replies
- 0 kudos
Export to Excel xlsx
Hi All Does anyone have some code or example of how to export my Databricks SQL results directly to an existing spreadsheet?Many ThanksKody_Devl
- 23716 Views
- 2 replies
- 0 kudos
- 0 kudos
There is an add-on directly import table to spreadsheet. https://workspace.google.com/marketplace/app/bricksheet/979793077657
- 0 kudos
| User | Count |
|---|---|
| 1611 | |
| 768 | |
| 348 | |
| 286 | |
| 252 |
