- 5602 Views
- 4 replies
- 1 kudos
Resolved! How to execute stored procedures on synapse sql pool from databricks
In the current version of databricks, previous methods to execute stored procedures seem to fail. spark.sparkContext._gateway.jvm.java.sql.DriverManager/spark._sc._gateway.jvm.java.sql.DriverManager returns that it is JVM dependent and will not work....
- 5602 Views
- 4 replies
- 1 kudos
- 1 kudos
can you create a connection to external data in unity catalog, and then:use <connectiondb>;exec <sp>
- 1 kudos
- 1342 Views
- 1 replies
- 0 kudos
playcasinosnj.com
One of the best platforms where you can find games like online slots is https://playcasinosnj.com/. It is in my opinion the safest site to find the games that suit you. I also had the chance to try out the different games and they all worked for me. ...
- 1342 Views
- 1 replies
- 0 kudos
- 0 kudos
Cool, I love slots. Thanks for the tip. I like that they are very diverse and easy to learn, so I often enjoy spending a couple of hours on these games in the evenings after work. Also, I recently found some betting apps not on Gamstop that turned ou...
- 0 kudos
- 1131 Views
- 1 replies
- 0 kudos
Peformnace improvement of Databricks Spark Job
Hi,I need performance improvement for data bricks job in my project. Here are some steps being done in the project1. Read csv/Json files with small size (100MB,50MB) from multiple locations in s32. Write the data in bronze layer in delta/parquet form...
- 1131 Views
- 1 replies
- 0 kudos
- 0 kudos
In case of performance issues, always look for 'expensive' operations. Mainly wide operations (shuffle) and collecting data to the driver.Start with checking how long the bronze part takes, then silver etc.Pinpoint where it starts to get slow, then d...
- 0 kudos
- 5691 Views
- 7 replies
- 0 kudos
WATER MARK ERROR WHILE JOINING WITH MULTIPLE STREAM TABLES
I am creating a ETL pipeline where i am reading multiple stream table into temp tables and at the end am trying to join those tables to get the output feed into another live table. So for that am using below method where i am giving list of tables as...
- 5691 Views
- 7 replies
- 0 kudos
- 0 kudos
it is necessary for the join so if the dataframe has a watermark that's enough.No need to define it multiple times.
- 0 kudos
- 18322 Views
- 1 replies
- 0 kudos
Passing variables from python to sql in a notebook using serverless compute
I've got a notebook that I've written that's going to execute some python code to parse the workspace id to figure out which of my environments that I'm in and set a value for it. I then want to take that value, and pass it through to a code block of...
- 18322 Views
- 1 replies
- 0 kudos
- 0 kudos
Thanks Kaniz, this is a great suggestion. I'll look into it and how it can work for my projects.
- 0 kudos
- 4417 Views
- 1 replies
- 0 kudos
Terminating cluster programmatically
Is there any python script that allows me to terminate (not delete) a cluster in the notebook, similar to this R equivalent ofterminate_cluster(cluster_id, workspace, token = NULL, verbose = T, ...)
- 4417 Views
- 1 replies
- 0 kudos
- 3919 Views
- 7 replies
- 2 kudos
Resolved! i created External database but unable to transferring table to Storage Acc(BLOBcontainer-Gold)
Hi , I done activities Bronze and Silver , after i trying to saving table to Gold container but unable to storing .i created External database .I want store data to PARQUET but not supporting ,only DELTA.only MANAGED LOCATION supporting but unabl...
- 3919 Views
- 7 replies
- 2 kudos
- 2686 Views
- 4 replies
- 0 kudos
DQ-Quality Check- what are the best method to validate the two parquet files .
DQ-Quality Check. we have to validate the data between landing data and bronze data with data quality . below are the data quality checks. 1. find the counts between the 2 files. if it is matched then go for 2 point.2. if counts are matched, then va...
- 2686 Views
- 4 replies
- 0 kudos
- 0 kudos
Try with this , this is for second point if first points already matches .# Define key columns key_columns = ["key_column1", "key_column2"] # Adjust according to your data schema # Perform an outer join to find mismatches joined_df = landing_df.ali...
- 0 kudos
- 3236 Views
- 1 replies
- 0 kudos
Upload files from Databricks to Google Drive
Is it possible to upload files from Databricks to Google Drive? How?
- 3236 Views
- 1 replies
- 0 kudos
- 0 kudos
@CaptainJack You can use Python + Google Drive API. Example: https://medium.com/the-team-of-future-learning/integrating-google-drive-api-with-python-a-step-by-step-guide-7811fcd16c44
- 0 kudos
- 5722 Views
- 0 replies
- 1 kudos
How to calculate the individual file count, file size and number of rows on a Delta table?
There are instances where we need to know the individual file size or file count present in the delta table rather than the average size. we can use the below query to determine that. %sql select count(*) as rows, file_path, file_size from (select * ...
- 5722 Views
- 0 replies
- 1 kudos
- 4532 Views
- 2 replies
- 0 kudos
How to get the files one by one in blob storage using pyspark/python
how to write the pyspark/python to get the files one by one in blob storage.
- 4532 Views
- 2 replies
- 0 kudos
- 0 kudos
@rameshybr # List files in a directory files = dbutils.fs.ls("/mnt/<mount-name>/path/to/directory") for file in files: file_path = file.path # Read each file into a DataFrame / if you file format is parquet for example i am taking df = s...
- 0 kudos
- 9107 Views
- 5 replies
- 7 kudos
How to change SQL editor / schema browser defalut catalog / database
On SQL editor / schema browser Is there a way to change the default catalog / database ? My mine always fixed on my unity catalog.
- 9107 Views
- 5 replies
- 7 kudos
- 7 kudos
Hi, From the dropdown you can get the data objects.https://docs.databricks.com/sql/user/queries/queries.html#browse-data-objects-in-sql-editorPlease let us know if this helps. Also, please tag @Debayan​ with your next comment so that I will get notif...
- 7 kudos
- 1659 Views
- 2 replies
- 0 kudos
isolated databricks cluster call from synapses or azure datafactory
https://learn.microsoft.com/en-us/answers/questions/1919424/isolated-databricks-cluster-call-from-synapses-orhow can I create a job in databricks with parameters of isolated from synapses or azure datafactory, because I can not find any option that a...
- 1659 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi warner thanks for your question, I share the link service in synapses updated, currently we have a pool in databricks then what we do with the link service is that it creates a job and uploads an instance with the resources of our pool but to uplo...
- 0 kudos
- 1652 Views
- 1 replies
- 0 kudos
Compute pools max capacity and ideal compute settings
Hi there,I'm having a difficult time understanding the compute side of our jobs under the hood, and I checked the documentation but don't have clear answers so far, so hopefully someone will provide some clarity.I set up pools to use for our overnigh...
- 1652 Views
- 1 replies
- 0 kudos
- 0 kudos
I moved away from pools since cluster reuse is possible in databricks jobs.Why? More control over your workers, no need to find a good waiting time and you can even run multiple tasks on a single cluster.Why your jobs fail is not clear to me.When no ...
- 0 kudos
- 1570 Views
- 1 replies
- 0 kudos
After catalog recreation I cannot create folders with names that previously had existed
I have a catalog with multiple schemas, tables and volumes. Each volume contains two folders with specific names that I cannot change.When I drop the whole catalog (CASCADE) and create catalog with the same name, with the same schemas, tables and vol...
- 1570 Views
- 1 replies
- 0 kudos
- 0 kudos
I also had issues a while back with (what I believe is) some kind of cache. I have no knowledge how to override this.What I did is what you tried: waiting.
- 0 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
