- 3420 Views
- 2 replies
- 0 kudos
count or toPandas taking too long
Hi,I am fetching data from unity catalog from notebooks using spark.sql(). The query takes just a few seconds - I am actually trying to retrieving 2 rows - but some operations like count() or toPandas() take forever. I wonder why does it take so long...
- 3420 Views
- 2 replies
- 0 kudos
- 0 kudos
Hey @jimcast how are you? You can check the internals and have a good hint of what's happening using the SparkUI. Filter and select the jobs that are taking the longest and check what is being requested on the SQL/Data Frame tab, as well as their pla...
- 0 kudos
- 1923 Views
- 2 replies
- 2 kudos
- 1923 Views
- 2 replies
- 2 kudos
- 2 kudos
Yes, storage-partitioned joins can be optimized for data skewness. Techniques like adaptive query processing and dynamic repartitioning help distribute the workload evenly across nodes. clipping path service provider By identifying and addressing dat...
- 2 kudos
- 2555 Views
- 3 replies
- 0 kudos
DataBricks Certification Exam Got Suspended. Require support for the same.
Hello Team, I encountered Pathetic experience while attempting my 1st DataBricks certification. Abruptly, Proctor asked me to show my desk, after showing he/she asked multiple times.. wasted my time and then suspended my exam without giving any reaso...
- 2555 Views
- 3 replies
- 0 kudos
- 0 kudos
@Kaniz @Cert-Team @Sujitha I have sent multiple emails to the Support team to reschedule my exam with Date, but I have not received any confirmation from them.Please look into this issue and reschedule the exam as soon as possible. This certification...
- 0 kudos
- 995 Views
- 0 replies
- 0 kudos
Got suspended which attempting Databricks certified Associate Developer for Apache Spark 3.0 Python
Hi Team, My Databricks Certified exam got suspended.I was continuously in front of the camera and an alert appeared and then my exam resumed. Then later a support person asked me to show the entire table and entire room, I have showed around the room...
- 995 Views
- 0 replies
- 0 kudos
- 5614 Views
- 1 replies
- 1 kudos
Resolved! capture return value from databricks job to local machine by CLI
Hi,I want to run a python code on databricks notebook and return the value to my local machine. Here is the summary:I upload files to volumes on databricks. I generate a md5 for local file. Once the upload is finished, I create a python script with t...
- 5614 Views
- 1 replies
- 1 kudos
- 1 kudos
Hello @pshuk, You could check the below CLI commands: get-run-output Get the output for a single run. This is the REST API reference, which relates to the CLI command: https://docs.databricks.com/api/workspace/jobs/getrunoutput export-run There's al...
- 1 kudos
- 3539 Views
- 1 replies
- 0 kudos
Resolved! Error Code: METASTORE_DOES_NOT_EXIST when using Databricks API
Hello, I'm attempting to use the databricks API to list the catalogs in the metastore. When I send the GET request to `/api/2.1/unity-catalog/catalogs` , I get this error I have checked multiple times and yes, we do have a metastore associated with t...
- 3539 Views
- 1 replies
- 0 kudos
- 0 kudos
Turns out I was using the wrong databricks host url when querying from postman. I was using my Azure instance instead of my AWS instance.
- 0 kudos
- 24546 Views
- 3 replies
- 4 kudos
Resolved! Use SQL Server Management Studio to Connect to DataBricks?
The Notebook UI doesn't always provide the best experience for running exploratory SQL queries. Is there a way for me to use SQL Server Management Studio (SSMS) to connect to DataBricks? See Also:https://learn.microsoft.com/en-us/answers/questions/74...
- 24546 Views
- 3 replies
- 4 kudos
- 4 kudos
What you can do is define a SQL endpoint as a linked server. Like that you can use SSMS and T-SQL.However, it has some drawbacks (no/bad query pushdown, no caching).Here is an excellent blog of Kyle Hale of databricks:Tutorial: Create a Databricks S...
- 4 kudos
- 3500 Views
- 1 replies
- 2 kudos
ingest csv file on-prem to delta table on databricks
Hi,So I want to create a delta live table using a csv file that I create locally (on-prem). A little background: So I have a working ELT pipeline that finds newly generated files (since the last upload), and upload them to databricks volume and at th...
- 3500 Views
- 1 replies
- 2 kudos
- 2 kudos
Hello @pshuk , Based on your description, you have an external pipeline that writes CSV files to a specific storage location and you wish to set up a DLT based on the output of this pipeline. DLT offers has access to a feature called Autoloader, whic...
- 2 kudos
- 2652 Views
- 3 replies
- 3 kudos
I am facing an issue while generating the DBU consumption report and need help.
I am trying to access the following system tables to generate a DBU consumption report, but I am not seeing this table in the system schema. Could you please help me how to access it?system.billing.inventory, system.billing.workspaces, system.billing...
- 2652 Views
- 3 replies
- 3 kudos
- 3235 Views
- 2 replies
- 0 kudos
Delta Sharing - Info about Share Recipient
What information do you know about a share recipient when they access a table shared to them via Delta Sharing?Wondering if we might be able to utilize something along the lines of is_member, is_account_group_member, session_user, etc for ROW and COL...
- 3235 Views
- 2 replies
- 0 kudos
- 0 kudos
Now that I'm looking closer at the share credentials and the recipient entity you would really need a way to know the bearer token and relate that back to various recipient properties - databricks.name and any custom recipient property tags you may h...
- 0 kudos
- 2948 Views
- 0 replies
- 0 kudos
Parallel kafka consumer in spark structured streaming
Hi,I have a spark streaming job which reads from kafka and process data and write to delta lake.Number of kafka partition: 100number of executor: 2 (4 core each)So we have 8 cores total which are reading from 100 partitions of a topic. I wanted to un...
- 2948 Views
- 0 replies
- 0 kudos
- 1735 Views
- 0 replies
- 1 kudos
how to develop Notebooks on vscode for git repos?
I am able to use vscode extension + databricks connect to develop Notebooks on my local computer and run them on my databricks cluster. However I can not figure out how to develop the Notebooks that have the file `.py` extension but identified by Dat...
- 1735 Views
- 0 replies
- 1 kudos
- 1905 Views
- 1 replies
- 0 kudos
Error While Running Table Schema
Hi All,I am facing issue while running a new table in bronze layer.Error - AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to alter table.com.databricks.backend.common.rpc.SparkDriverExceptions$SQLExecutionException: org.a...
- 1905 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @Mirza1 , Could you please share the source code that is generating the exception, as well as the DBR version you are currently using? This will help me better understand the issue.
- 0 kudos
- 3616 Views
- 1 replies
- 0 kudos
Resolved! How does coalesce works internally
Hi Databricks team,I am trying to understand internals of spark coalesce code(DefaultPartitionCoalescer) and going through spark code for this. While I understood coalesce function but I am not sure about complete flow of code like where its get call...
- 3616 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @subham0611 , The coalesce operation triggered from user code can be initiated from either an RDD or a Dataset, with each having distinct codepaths: RDD: https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/rdd/RDD...
- 0 kudos
- 9613 Views
- 2 replies
- 0 kudos
Resolved! Why saving pyspark df always converting string field to number?
import pandas as pd from pyspark.sql.types import StringType, IntegerType from pyspark.sql.functions import col save_path = os.path.join(base_path, stg_dir, "testCsvEncoding") d = [{"code": "00034321"}, {"code": "55964445226"}] df = pd.Data...
- 9613 Views
- 2 replies
- 0 kudos
- 0 kudos
@georgeyjy Try opening the CSV as text editor. I bet that Excel is automatically trying to detect the schema of CSV thus it thinks that it's an integer.
- 0 kudos
