- 5226 Views
- 2 replies
- 0 kudos
Databricks setup/deployment checklist/best practices
Hi Team, could you please share or guide us on any checklist/best practices for Databricks setup/deployment?
- 5226 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Phani1 , here are some best practices https://github.com/Azure/AzureDatabricksBestPractices/tree/master and you could take these points as your "checklist".Choose the right Databricks Workspace:Decide on the appropriate Azure region for your Data...
- 0 kudos
- 3454 Views
- 1 replies
- 0 kudos
Resolved! Read file with Delta Live Tables from external location (Unity Catalog)
As far as I understand, Delta Live Tables should now support reading data from an external location, but I can’t get it to work. I’ve added an ADLS container to Unity Catalog as an external location. There’s a folder in the container containing an ex...
- 3454 Views
- 1 replies
- 0 kudos
- 0 kudos
I misspelled the folder name; I got it working now The error message could have been more informative
- 0 kudos
- 1530 Views
- 0 replies
- 0 kudos
Highly Performant Data Ingestion and Processing Pipelines
Hi everyone,I am working on a project that requires highly performant pipelines for managing data ingestion, validation, and processing large data volumes from IOT devices.I am interested in knowing:- The best way to ingest from EventHub/Kafka sinks-...
- 1530 Views
- 0 replies
- 0 kudos
- 3568 Views
- 2 replies
- 1 kudos
Databricks Certified Associate Developer for Apache Spark 3.0 - Python sample paper
I am trying to get sample papers for Databricks Certified Associate Developer for Apache Spark 3.0 - Python before I attempt my exam
- 3568 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi, there, This exam has a Practice Exam, I think that's what you are looking for.
- 1 kudos
- 1300 Views
- 1 replies
- 0 kudos
Infrastructure reverse engineering
How can we do infrastructure reverse engineering in Databricks? Any kind of solution accelerator or tool that can scan through the environment and identify the solutions /components that are used in the Databricks environment and the complexity withi...
- 1300 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi Team,Could you please help us on this.Regards,Phanindra
- 0 kudos
- 1842 Views
- 1 replies
- 0 kudos
Got Suspended while taking Databricks Certified Data Analyst Associate Assessment
Hi Team, What I experienced today from the proctor was so not nice, my experience with proctor today was very frustrating and pathetic, I was taking my assessment today 11-26-2023, 2pm, I was already on between question 22nd to 25th when my assessmen...
- 1842 Views
- 1 replies
- 0 kudos
- 0 kudos
@osalawu Sorry to hear you had an issue with your exam. In order to protect your Webassessor account information, please file a ticket with our support team. Please include your Webassessor login ID, the exam, and a couple of dates and times that wil...
- 0 kudos
- 1568 Views
- 0 replies
- 0 kudos
Unable to log metrics to table/driver using spark StreamingQueryListener
Hi Everyone,Hope everyone is enjoying their holiday time.I have spark streaming job (copy activity from location A to location B) and a Query Listener and attached to the streaming job.As mentioned over the attachments I can see numInputRows gets log...
- 1568 Views
- 0 replies
- 0 kudos
- 2709 Views
- 0 replies
- 0 kudos
Arguments parsing in Databricks python jobs
On Databricks created a job task with task type as Python script from s3. However, when arguments are passed via Parameters option, running into unrecognized arguments' error.Code in s3 file:import argparse def parse_arguments(): parser = argpar...
- 2709 Views
- 0 replies
- 0 kudos
- 1695 Views
- 0 replies
- 0 kudos
Unity Catalog - Limited Options for Connection Objects
I’m currently trying to create a Foreign Catalog based on a Connection object of type SQLSERVER. This would allow me to directly access our on-premises MS SQL database from within Azure Databricks using Unity Catalog.As I’m part of a large organizati...
- 1695 Views
- 0 replies
- 0 kudos
- 1630 Views
- 1 replies
- 0 kudos
Can not able to connect to ADLS Gen2 using ABFSS (urgent)
Hi all,I was having an error when trying to connect to azure data lake. "SSLHandshakeExceptio" Please see the detail below.Please help!
- 1630 Views
- 1 replies
- 0 kudos
- 0 kudos
The error message you provided, "SSLHandshakeExceptio," indicates that there is a problem with the SSL handshake between your system and the Azure Data Lake. This issue can be caused by various factors, such as incorrect settings, network issues, or ...
- 0 kudos
- 2072 Views
- 0 replies
- 0 kudos
Structured Streaming - Kafka Offset Management
In my team, we decided to move from spark streaming to structured streaming, mainly cause it says that it's legacy and we want to benefit new features from structured streaming.However we have an issue with committing offsets.Previously on spark stre...
- 2072 Views
- 0 replies
- 0 kudos
- 21883 Views
- 2 replies
- 0 kudos
.overwriteschema + writestream
HelloI have issue with overwriting schema while using writestream - I do not receive any error - however schema remain unchangedBelow exampledf_abc = spark.readstream .format("cloudFiles") .option("cloudFiles.format", "parquet") .option"cloudF...
- 21883 Views
- 2 replies
- 0 kudos
- 10404 Views
- 3 replies
- 0 kudos
dataset.cache() not working : NoSuchObjectException(message:There is no database named global_temp)
ERROR RetryingHMSHandler: NoSuchObjectException(message:There is no database named global_temp)at org.apache.hadoop.hive.metastore.ObjectStore.getMDatabase(ObjectStore.java:508)at org.apache.hadoop.hive.metastore.ObjectStore.getDatabase(ObjectStore.j...
- 10404 Views
- 3 replies
- 0 kudos
- 1543 Views
- 0 replies
- 0 kudos
I want to perform transpose in my dataframe using pyspark
I have 26 columns and 18k rows which I want to transpose and make 18k columns and 26 rows.Don't want any data summation of aggregation just transpose as it is. Can anyone pls suggest.P.S Not via converting to pandas because of large data
- 1543 Views
- 0 replies
- 0 kudos
- 2589 Views
- 1 replies
- 0 kudos
Delta Live Tables Slowly Changing Dimensions Type 2 with Joins
Hi,I may be missing something really obvious here. The organisation I work for has started using Delta Live Tables in Databricks for data modelling, recently. One of the dimensions I am trying to model takes data from 3 existing tables in our data la...
- 2589 Views
- 1 replies
- 0 kudos
- 0 kudos
Can it be because the default join is `inner` and that means the row must exists in both tables
- 0 kudos
