- 83152 Views
- 13 replies
- 4 kudos
How can I export a result of a SQL query from a databricks notebook?
The "Download CSV" button in the notebook seems to work only for results <=1000 entries. How can I export larger result-sets as CSV?
- 83152 Views
- 13 replies
- 4 kudos
- 4 kudos
If you have a large dataset, you might want to export it to a bucket in parquet format from your notebook:%python df = spark.sql("select * from your_table_name") df.write.parquet(your_s3_path)
- 4 kudos
- 5726 Views
- 4 replies
- 3 kudos
Sending email alerts to non-databricks user
I am trying to send email alerts to a non databricks user. I am using Alerts feature available in SQL. Can someone help me with the steps.Do I first need to first add Notification Destination through Admin settings and then use this newly added desti...
- 5726 Views
- 4 replies
- 3 kudos
- 3 kudos
Hi @Mitali Lad Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers yo...
- 3 kudos
- 1850 Views
- 1 replies
- 0 kudos
integrating Azure Databricks with AAD
Hi Team, Could you please provide the details/process for integrating Azure Databricks - Unity Catalog and AAD? Regards,Phani
- 1850 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @Phani1 ,These doc pages might be useful for you: Set up and manage Unity CatalogSync users and groups from Microsoft Entra ID
- 0 kudos
- 2221 Views
- 1 replies
- 1 kudos
Upsert into a Delta Lake table with merge when using row masking function
I'm using databricks rls functions on my tables, and i need to make some merges into, but tables with rls functions does not support merge operations (https://docs.databricks.com/en/data-governance/unity-catalog/row-and-column-filters.html#limitation...
- 2221 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @ismaelhenzel, if you want to use the "MERGE INTO" sql command, you must turn-off rls. This is by design.
- 1 kudos
- 1472 Views
- 1 replies
- 0 kudos
Delay rows coming into DLT pipeline
Backgroundand requirements: We are reading data from our factory and storing it in a DLT table called telemetry with columns sensorid, timestamp and value. We need to get rows where sensorid is “qrreader-x” and join with some other data from that sam...
- 1472 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Mathias, I'd say that watermarking might be a good solution for your use case. Please check Control late data threshold with multiple watermark policy in Structured Streaming. If you want to dig-in further there's also: Spark Structured Streami...
- 0 kudos
- 2040 Views
- 1 replies
- 1 kudos
Optimizing Data Insertion Speed for JSON Files in DLT Pipeline
Background:I'm working on a data pipeline to insert JSON files as quickly as possible. Here are the details of my setup: File Size: 1.5 - 2 kB eachFile Volume: Approximately 30,000 files per hourPipeline: Using Databricks Delta Live Tables (DLT) in c...
- 2040 Views
- 1 replies
- 1 kudos
- 2888 Views
- 1 replies
- 0 kudos
Databricks Run Notebook GitHub Action
The GitHub action databricks/run-notebook to deploy and run a notebook from GitHub to DBX awaits the completion of the Job. The pulsar streaming job that I have is a long running job due to which the Action times out when the access token it uses to ...
- 2888 Views
- 1 replies
- 0 kudos
- 0 kudos
https://github.com/databricks/run-notebook/issues/53#issue-2321682696
- 0 kudos
- 3737 Views
- 3 replies
- 2 kudos
Is there a way to add a date parameter to the jobs run API call?
Hi there I am currently making a call to the Databricks API jobs run endpoint. I would like to make this call on a daily basis to get data on the jobs run in the past 24 hours and add this to my delta table. Is there a way to set a GTE value in the A...
- 3737 Views
- 3 replies
- 2 kudos
- 2 kudos
Actually the "start_time_to" parameter doesn't seem to work at all. Neither alone or together with "start_time_from" (Whenever used the api call returns nothing). I'd like to report this as an issue as we want to automate our cluster usage monitoring...
- 2 kudos
- 3699 Views
- 3 replies
- 0 kudos
Why Pandas on Spark can trigger `Driver is up but is not responsive, likely due to GC` ?
I am using the distributed Pandas on Spark, not the single node Pandas.But when I try to run the following code to transform a data frame with 652 x 729803 data points df_ps_pct = df.pandas_api().pct_change().to_spark() , it returns me this error: ...
- 3699 Views
- 3 replies
- 0 kudos
- 0 kudos
@Hi @Dicer I don't think you have a problem with the workers, since you are running distributed Pandas, work is going to be paralleled either way. When the data is collected back to the Driver, then it might be overloaded (since the Driver has to co...
- 0 kudos
- 7641 Views
- 5 replies
- 3 kudos
Resolved! Unit Testing with the new Databricks Connect in Python
I would like to create a regular PySpark session in an isolated environment against which I can run my Spark based tests. I don't see how that's possible with the new Databricks Connect. I'm going in circles here, is it even possible?I don't want to ...
- 7641 Views
- 5 replies
- 3 kudos
- 3 kudos
Given this doesn't work on serverless compute, aren't those tests very slow to complete due to the compute startup time? I'm trying to steer away from databricks connect for unit testing for this reason. If they supported serverless, that would be a ...
- 3 kudos
- 4242 Views
- 1 replies
- 1 kudos
SQL Sub Query Not Working
Olá pessoal, espero que todos estejam bem!Por favor, você pode ajudar? ou orientação?Está retornando um erro na instrução "CASE WHEN"A lógica é a seguinte: Se o campo `valor` FOR NULO ENTÃO eu executo uma Sub Consulta usando os filtros: origem, desti...
- 4242 Views
- 1 replies
- 1 kudos
- 1 kudos
Olá @opl12 , tudo bem?Se possível, poderia enviar a mensagem de erro completa? Aparecerá uma lista separada por vírgulas de possíveis colunas logo após "Did you mean one of the following? [...]".Poderia testar sem os `` e me informar o resultado? Bes...
- 1 kudos
- 3506 Views
- 1 replies
- 0 kudos
Parallel read of many delta tables
I need to read many delta tables in azure object storage (block blobs). There is no root object delta table, but rather many fragmented delta tables that share a common schema but not common paths.Iterating over the paths with a for loop is performin...
- 3506 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @leobocci , In order to read multiple Delta tables, multiple read operations are required. You can trigger the read operations simultaneously through the Job Workflows, DLT, Databricks CLI, DBSQL, Interactive Clusters and other resources.If the...
- 0 kudos
- 5038 Views
- 3 replies
- 1 kudos
How to Programmatically Retrieve Cluster Memory Usage?
Hi!I need help with the following:Programmatically retrieve the maximum memory configured for the cluster attached to the notebook/job - I think this is achievable through the system tables or Clusters API, but I'm open to other suggestionsExecute a ...
- 5038 Views
- 3 replies
- 1 kudos
- 1 kudos
Great use case! Have you ever heard about Prometheus with Spark 3.0? Its a tool that can export live metrics for your jobs and runs which writes to a sink where you can read with a stream. I've personally never used in such use case, but there you ca...
- 1 kudos
- 4004 Views
- 1 replies
- 0 kudos
Resolved! databricks cli create job
Hi, using the Databricks cli, I exported the jobs in json format from the workspace in Azure, using the same json to create a new job, but in a workspace in AWS, the error below occurs.To create a job via Databricks cli on AWS, do you need to change ...
- 4004 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi, I already found the error, you need to use @ in the path.Thanks.
- 0 kudos
- 17137 Views
- 5 replies
- 1 kudos
Move whole workflow from Dev to Prod
I have a workflow created in Dev, now I want to move the whole thing to prod and schedule it. The workflow has multiple notebooks, dependent libraries, parameters and such. How to move the whole thing to prod, instead of moving each notebooks and rec...
- 17137 Views
- 5 replies
- 1 kudos
- 1 kudos
I ended up creating a python script to just do the export, here is the code below.It will match on Job name, if it matches it will update otherwise it will import. import requests source_token = '' source_instance = 'adb-000000000000000.00.azuredata...
- 1 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
