Hi, i have a problem with my on-premise sql conection from databricks.i have my python code wich use pymssql and it is working well but my spark code doest not and i am using the same credential for both, my spark code is """# Read data from SQL Serv...
I've ran a dual multiprocessing and multithreading solution in python before using the multiprocessing and concurrent futures python modules. However, since the multiprocessing module only runs on the driver node, I have to instead use sc.parallelize...
Just wonder what is difference if we declare primary key as binary data type vs string data type
I am trying to log in to my workspace, but it takes a very long time to evaluate. Sometimes, it simply fails to do so, and I am prompted with the below message. {"error_code":"TEMPORARILY_UNAVAILABLE","message":"Authentication is temporarily unavaila...
Hi Menotron.Eventually, I realized it myself. Thank you for your comment
Hi All,Recently i created a custom docker image to run on databricks on cluster, Image built successfully and also cluster started but when i try to run anything in the notebook it's throwing error:File "/databricks/python_shell/scripts/db_ipyke...
Is there any way to check job cluster is unity catalog enabled or not in databricks using python.I tried with jobs api https://{host_name}/api/2.0/jobs/get?job_id={job_id}, but I didn't that cluster is unity catalog enabled or not.Could anyone sugges...
To check if a job cluster is Unity catalog enabled in Databricks programmatically using Python, you can use the Databricks REST API. Here's an example of how you can do it:Import the required modules:import requestsSet up the necessary variables:host...
Preparing for databricks eligibility!Is the content below correct?"If the queries are running sequentially then scale up (increase the size of the cluster from 2x small to 4x large)If the queries are running concurrently or with many users then scale...
Scaling in Databricks involves two aspects: vertical scaling (scale up) and horizontal scaling (scale out). Vertical Scaling (Scale Up): If your queries are running sequentially, meaning one query at a time, and you want to improve performance for a...
Hi,I'm trying to assign a location to a new database in Databricks SQL. Normally I'd do this in Python since we specify storage account names from secret scopes, however I'm attempting to do all of this from a SQL warehouse. When doing this I seem to...
Hello @kellybe ,CREATE DATABASE IF NOT EXISTS new_database LOCATION format_string('abfss://container-name@%s.dfs.core.windows.net/', select SECRET('secret-scope', 'storage-account-name')); Add Select before secert
Hi,Is there an example of incorporating Databricks Feature Store into DLT pipelines? Is this possible natively via a Python notebook part of the pipeline (FYI - docs say needs ML Runtime?). If not completely DLT-able, what is the best current way t...
Hi!!Can someone tell me how to connect the confluent cloud to Databricks? I am new to this so please elaborate on your answer.
You might want to watch this as well https://www.confluent.io/resources/online-talk/innovate-faster-and-easier-with-confluent-and-databricks-on-azure/?utm_medium=sem&utm_source=google&utm_campaign=ch.sem_br.nonbrand_tp.prs_tgt.dsa_mt.dsa_rgn.india_ln...
I am really confused about the DEFAULT capability of Databricks SQL. I looked at the documentation for the minimum required DBR to get the capability yet we still need to enable it as a table property? I updated my cluster's DBR from 12.2 to 13.1.Any...
Hello @Gim, Got the same problem. Tried with the instruction "GENERATED ALWAYS AS (CAST(CURRENT_DATE() AS DATE))" but code is returning "Error in SQL statement: DeltaAnalysisException: current_date() cannot be used in a generated column" If you find ...
I am accessing an on premise SQL Server table. The table is relatively small (10 000 rows), and I access it usingspark.read.jdbc(url=jdbcUrl, table = query)Every day there are new records in the on prem table that I would like to append in my bronze ...
As I said, there is no unique identifier in the table that would allow me to do any sort of Join between my source table and my bronze table.
Challenges in query long run time; what are the recommended steps to improve performance
The question needs more precision : is it the cluster startup that takes a while ? If yes, try serverless warehousesAre there many queries running in parallel and that is where you see a slow down ? Each cluster can only run 10 queries in parallel, s...
How can I optimize the cost on our Databricks platform ? Despite some optimization actions I've taken so far it's still difficult to lower the cost. I tried different technics like Vacuum , or shutting down a cluster running after 30 mins but still d...
Make sure you're using a cluster that is the right size for your workload. You can greatly reduce the costs by using smaller clusters.
Greetings community, I am new to using databricks and for some time I have tried some scripts in notebook. I would like your help on a task: Carry out a personalized mailing where, First, a query of the number of records in the test table is performe...
| User | Count |
|---|---|
| 1613 | |
| 771 | |
| 349 | |
| 286 | |
| 253 |
