- 1592 Views
- 1 replies
- 0 kudos
Databricks certificate exam got suspended, Need help to reschedule the exam
Hi @Cert-Team ,Today Mu Databricks certificate exam was suspended in the midway after approx 1 hour. Proctor asked me to show the room and after sometime he conveyed that my exam is being suspended and i will have to reach out to Databricks team for ...
- 1592 Views
- 1 replies
- 0 kudos
- 0 kudos
@Ajaykumar the support team has responded to your ticket.
- 0 kudos
- 3736 Views
- 6 replies
- 0 kudos
Getting an error while editing databrics metastore "databricks_metastore" using terraform (Azure)
Hi , I am getting an error (Metastore has reached the limit for metastores in region ) while updating metatsore using terraform , below the script i am using for updating metastore . There is already a metastore available i dont want to create the ne...
- 3736 Views
- 6 replies
- 0 kudos
- 0 kudos
Hi, Thank you for the suggestions, But i am still getting same error .
- 0 kudos
- 2406 Views
- 4 replies
- 0 kudos
Help to reschedule of my exam
Hi team,My Databricks Certified Data Engineer Associate exam got suspended within 40 minutes. I had also shown my exam room to the proctor. My exam got suspended due to watch. I was not using anything. Some questions are so big in the exam so I had t...
- 2406 Views
- 4 replies
- 0 kudos
- 3095 Views
- 1 replies
- 0 kudos
How to Resolve ConnectTimeoutError When Registering Models with MLflow
Hello everyone,I'm trying to register a model with MLflow in Databricks, but encountering an error with the following command: model_version = mlflow.register_model(f"runs:/{run_id}/random_forest_model", model_name) The error message is as follows:...
- 3095 Views
- 1 replies
- 0 kudos
- 0 kudos
@otara_geni if you are still struggling, try this - set the environmental variable in your code just before logging the model with the URL of the regional S3 endpoint (from the error it looks like MLFlow is attempting to use global one, which may not...
- 0 kudos
- 13435 Views
- 12 replies
- 2 kudos
Getting HTML sign I page as api response from databricks api with statuscode 200
Response:<!doctype html><html><head> <meta charset="utf-8" /> <meta http-equiv="Content-Language" content="en" /> <title>Databricks - Sign In</title> <meta name="viewport" content="width=960" /> <link rel="icon" type="image/png" href="...
- 13435 Views
- 12 replies
- 2 kudos
- 2 kudos
Hey @schunduri, not entirely sure because our SRE did the change, but the machine the pipeline runs on must be within the same vnet as your DBKS workspace. If you need more guidance, I could try and check what we did but our SRE left the company sinc...
- 2 kudos
- 4361 Views
- 3 replies
- 0 kudos
What exactly is Vectorized query processing and columnar acceleration
Hey folks! I want to know and understand while using photon acceleration, there is a feature called columnar acceleration which basically is a method of storing data in columns rather than rows, which is particularly advantageous for analytical datab...
- 4361 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @szymon_dybczak, Thanks for reaching out! Please review the response and let us know if it answers your question. Your feedback is valuable to us and the community. If the response resolves your issue, kindly mark it as the accepted solution. This...
- 0 kudos
- 6265 Views
- 3 replies
- 1 kudos
How do I create spark.sql.session.SparkSession?
When I create a session n Databricks it is defaulting to spark.sql.connect.session.SparkSession. How can I connect to spark with out spark connect?
- 6265 Views
- 3 replies
- 1 kudos
- 1 kudos
Is there any solution to this? Pandera, Evidently and Ydata Profiling break because they don't speak a sql.connect session object. They expect a spark.sql.session.SparkSession it's very frustrating not being to use any of these libraries with the new...
- 1 kudos
- 4575 Views
- 2 replies
- 1 kudos
Structured Streaming from a delta table that is a dump of kafka and get the latest record per key
I'm trying to use Structured Streaming in scala to stream from a delta table that is a dump of a kafka topic where each record/message is an update of attributes for the key and no messages from kafka are dropped from the dump, but the value is flatt...
- 4575 Views
- 2 replies
- 1 kudos
- 1 kudos
I am confused about this recommendation. I thought the use of the append output mode in combination with aggregate queries is restricted to queries for which the aggregation is expressed using event-time and it defines a watermark.Could you clarify ?
- 1 kudos
- 5140 Views
- 3 replies
- 1 kudos
Resolved! Best Approach for Handling ETL Processes in Databricks
I am currently managing nearly 300 tables from a production database and considering moving the entire ETL process away from Azure Data Factory to Databricks.This process, which involves extraction, transformation, testing, and loading, is executed d...
- 5140 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi,Instead of 300 individual files or one massive script, try grouping similar tables together. For example, you could have 10 scripts, each handling 30 tables. This way, you get the best of both approches—This way you will have a freedom of easy deb...
- 1 kudos
- 3426 Views
- 4 replies
- 3 kudos
Understanding Flight Cancellations and Rescheduling in Airlines Using Databricks and PySpark
In the airline industry, it’s important to manage flights efficiently. Knowing why flights get canceled or rescheduled helps improve customer satisfaction and operational performance. In this article, I’ll show you how to use Databricks and PySpark t...
- 3426 Views
- 4 replies
- 3 kudos
- 12127 Views
- 8 replies
- 0 kudos
Unable to reactive an inactive user
Hi all,I am facing an issue with reactivating an inactive user i tried the following json with databricks cli run_update = { "schemas": [ "urn:ietf:params:scim:api:messages:2.0:PatchOp" ], "Operations": [ { "op": "replace", "path": "ac...
- 12127 Views
- 8 replies
- 0 kudos
- 0 kudos
@FunkybunchOO Thank you for your response! I will look into other connections, but we are not currently using SCIM. There must be something similar blocking the activation.
- 0 kudos
- 989 Views
- 0 replies
- 0 kudos
UCX
Hey folks! I want to know what are the features that UCX does not provides in UC or specially Hive to UC Migration that can be done manually but not using UCX. As UCX is currently in developing mode so there are so many drawbacks, can someone share t...
- 989 Views
- 0 replies
- 0 kudos
- 777 Views
- 0 replies
- 0 kudos
databricks billing
Is there a way to use a business checking account to pay for Databricks services?
- 777 Views
- 0 replies
- 0 kudos
- 2313 Views
- 2 replies
- 0 kudos
Resolved! Translating XMLNAMESPACE in SQL Databricks
We are loading a data source that contains XML. I am translating their queries to create views in Databricks. They use 'XMLNAMESPACES' to construct/parse XML. Below is an example. What is best practice for translating 'XMLNAMESPACES' in Databricks?...
- 2313 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @TinaN, To handle XMLNAMESPACES in Databricks, use the from_xml function for parsing XML data, where you can define namespaces within your parsing logic. Start by reading the XML data using spark.read.format("xml"), then apply the from_xml functio...
- 0 kudos
- 1214 Views
- 1 replies
- 0 kudos
Can I load the files based on the data in my table as variable without iterating through each row?
Hi,I have created this table which contains the data that I need for my source path and target table. source_path: /data/customer/sid={sid}/abc=1/attr_provider={attr_prov}/source_data_provider_code={src_prov}/So basically, the value of each row are c...
- 1214 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @zll_0091, To efficiently load only the necessary files without manually iterating through each row of your table, you can use Spark's DataFrame operations. First, read your table into a DataFrame and determine the maximum key value. Then, filter ...
- 0 kudos
