- 5359 Views
- 5 replies
- 4 kudos
Resolved! Generate longer token for Databricks with Azure.
I'm using DefaultAzureCredential from azure-identity to connect to Azure with service principal environment variables (AZURE_CLIENT_SECRET, AZURE_TENANT_ID, AZURE_CLIENT_ID).I can get_token from a specific scope for databricks like this:from azure.id...
- 5359 Views
- 5 replies
- 4 kudos
- 4 kudos
I made up an alternative solution. I made up my own python class to handle my PAT from Databricks : https://stackoverflow.com/questions/75071869/python-defaultazurecredential-get-token-set-expiration-or-renew-token/You can be fancier or even register...
- 4 kudos
- 4251 Views
- 5 replies
- 3 kudos
Generate sh2 hashkey while loading files to delta table
I have files in azure data lake. I am using autoloader to read the incremental filesfiles don't have primary key to load, In this case i want to use some columns and generate an hashkey and use it as primary key to do changes.In this case i want to ...
- 4251 Views
- 5 replies
- 3 kudos
- 3 kudos
Hi @Vijay Kumar J(Customer) , We haven’t heard from you since the last response from @Debayan Mukherjee and @Jordan Fox and I was checking back to see if his suggestions helped you. Or else, If you have any solution, please do share that with the...
- 3 kudos
- 1737 Views
- 2 replies
- 1 kudos
base64 encode is not matching with Oracle's base64 encode
Hi , base64 encode is not matching with Oracle's base64 encode. please see below result. Could anyone help me on this?In Azure Databricks: encoded= base64.b64encode(b'952B8D04E5CFB9BE')output is - b'OTUyQjhEMDRFNUNGQjlCRQ=='In Oracle: select utl_enco...
- 1737 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Dinu Sukumara, We haven’t heard from you since the last response from @Daniel Sahal, and I was checking back to see if his suggestions helped you.Or else, If you have any solution, please do share that with the community as it can be helpful to...
- 1 kudos
- 3274 Views
- 5 replies
- 2 kudos
How to replace LF and replace with ' ' in csv UTF-16 encoded?
I have tried several code and nothing worked. An extra space or line LF is going to next row in my output. All rows are ending in CRLF, but some rows end in LF and while reading the csv, it is not giving correct output. My csv have double dagger as d...
- 3274 Views
- 5 replies
- 2 kudos
- 2 kudos
Hi @shamly pt(Customer) , We haven’t heard from you since the last response from @sherbin w and @Ratna Chaitanya Raju Bandaru, and I was checking back to see if their suggestions helped you.Or else, If you have any solution, please do share th...
- 2 kudos
- 1965 Views
- 2 replies
- 1 kudos
DB Connect failing
For some reason, my dbconnect failed and I haven't been able to resolve the issue. I am connecting to an enterprise server. I was getting the following errors which (I believe) are now resolved.I defined the PYSPARK_DRIVER_PYTHON and PYSPARK_PYTHON v...
- 1965 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Paras Patel(Customer) , We haven’t heard from you since the last response from @Debayan Mukherjee, and I was checking back to see if his suggestions helped you. Or else, If you have any solution, please do share that with the community as it c...
- 1 kudos
- 2702 Views
- 4 replies
- 3 kudos
- 2702 Views
- 4 replies
- 3 kudos
- 3 kudos
Hi @Akanksha Kumari , We haven’t heard from you on the last response from @Mark Ferguson and @Hubert Dudek, and I was checking back to see if their suggestions helped you. Or else, If you have any solution, please do share that with the community ...
- 3 kudos
- 2984 Views
- 3 replies
- 2 kudos
implement autoloader to ingest data into delta lake, i have 100 different tables with full load , append merge senarios
i want to implement autoloader to ingest data into delta lake from 5 different source systems and i have 100 different tables in each database how do we dynamically address this by using autoloader , trigger once option - full load , append merge sen...
- 2984 Views
- 3 replies
- 2 kudos
- 2 kudos
Hi @Anil Kovilakar(Customer) , We haven’t heard from you on the last response from @Daniel Sahal and @Jordan Fox, and I was checking back to see if their suggestions helped you. Or else, If you have any solution, please do share that with the c...
- 2 kudos
- 2273 Views
- 2 replies
- 1 kudos
Ganglia not working with custom container services
Setup:custom docker container starting from the "databricksruntime/gpu-conda:cuda11" base image layer10.4 LTS (includes Apache Spark 3.2.1, Scala 2.12)multi-node, p3.8xlarge GPU computeWhen I try to view Ganglia metrics I am met with "502 Bad Gatewa...
- 2273 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @James W (Customer) , We haven’t heard from you since the last response from @Vivian Wilfred, and I was checking back to see if his suggestions helped you. Or else, If you have any solution, please do share that with the community as it can be ...
- 1 kudos
- 2348 Views
- 4 replies
- 9 kudos
one of the date datatype format issue in pysaprk
if anyone has encountered this date type format - 6/15/25 12:00 AM could you mention the right formatting to be used in Pyspark.Thanks in advance!
- 2348 Views
- 4 replies
- 9 kudos
- 9 kudos
Hi @KVNARK ., We haven’t heard from you since the last response from @Uma Maheswara Rao Desula and @Hubert Dudek I was checking back to see if his suggestions helped you. Or else, If you have any solution, please do share that with the community a...
- 9 kudos
- 6336 Views
- 4 replies
- 2 kudos
Resolved! slow Fetching results by client in databricks SQL calling from Azure Compute Instance (AML)
I'm using `databricks-sql-connector` in python3.8 to connect to an Azure SQL Wharehouse inside an Azure Machine Learning Compute Instance.I have this large result query, looking at the `query history` I check the time spent on doing the query, and se...
- 6336 Views
- 4 replies
- 2 kudos
- 2 kudos
So I made some few tests. Since you said that the Databricks SQL driver wasn't made to retrieve that amount of data. I went on Spark.I fired up a small spark cluster, the query was as fast as on SQL Warehouse, then I did a df.write.parquet("/my_path/...
- 2 kudos
- 2182 Views
- 4 replies
- 0 kudos
I am trying to write a data frame to Kafka topic with Avro schema for key and value using a schema registry URL. The to_avro function is not writing t...
I am trying to write a data frame to Kafka topic with Avro schema for key and value using a schema registry URL. The to_avro function is not writing to the topic and throwing an exception with code 40403 something. Is there an alternate way to do thi...
- 2182 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi,Could you please refer to https://github.com/confluentinc/kafka-connect-elasticsearch/issues/59 and let us know if this helps.
- 0 kudos
- 3047 Views
- 5 replies
- 2 kudos
SQL warehouse failing to start ( Please check network connectivity from the data plane to the control plane )
Hi, My SQL warehouse is failing to start with the following error message:Details for the latest failure: Error: [id: InstanceId(i-01b84b6705ff09104), status: INSTANCE_INITIALIZING, workerEnvId:WorkerEnvId(workerenv-3023557811934763-c8cef827-a038-455...
- 3047 Views
- 5 replies
- 2 kudos
- 2 kudos
Hi, There is a line in the attached logs as below:[Bootstrap Event] Can reach ohio.cloud.databricks.com: [FAILED][Bootstrap Event] DNS output for databricks-prod-artifacts-us-east-2.s3.us-east-2.amazonaws.com: Server: 10.187.0.2Address: 10.187.0.2#5...
- 2 kudos
- 2229 Views
- 3 replies
- 2 kudos
How to delete duplicate tables?
Hi Everyone,Accidently imported duplicate tables, guide me how to delete themusing data bricks community edition
- 2229 Views
- 3 replies
- 2 kudos
- 2 kudos
Hi @Mahesh Babu Uppala (Customer) , We haven’t heard from you on the last response from @Uma Maheswara Rao Desula and @Ratna Chaitanya Raju Bandaru , and I was checking back to see if their suggestions helped you. Or else, If you have any solu...
- 2 kudos
- 4132 Views
- 7 replies
- 8 kudos
Resolved! How can we move the excel file from adls to share point
I have one excel file in the adls, i want to move that file into sharepoint, but i tried this method in data factory, but in sink sharepoint is not available, is there any possible way to do this?
- 4132 Views
- 7 replies
- 8 kudos
- 8 kudos
Hi @Mohammed sadamusean(Customer) , We haven’t heard from you on the last response from @Daniel Sahal and @KVNARK ., and I was checking back to see if their suggestions helped you. Or else, If you have any solution, please do share that with th...
- 8 kudos
- 2929 Views
- 5 replies
- 5 kudos
Databricks Academy webpage showing insecure connection (in Chrome)
When I was trying to visit the Databricks Academy website https://customer-academy.databricks.com, it showed insecure connection as below.This happened at 8 January 2023 (AEDT) around 12:30pm.
- 2929 Views
- 5 replies
- 5 kudos
- 5 kudos
Hi @Selina Li, Thank you for reaching out!Let us look into this for you, and we'll circle back with an update.
- 5 kudos