Databricks Community

Access file location problem

import pandas as pd from apiclient.discovery import build from oauth2client.service_account import ServiceAccountCredentials df = spark.read.json("/FileStore/tables/cert.json")   SCOPES = ['https://www.googleapis.com/auth/analytics.readonly'] KEY_FIL...

RStudio on Databricks user experience

Is anybody actually using the RStudio app integration on Databricks? I'm surprised to find so little discussion in this forum. My team has been using it for about 3 months and it seems under-developed.1) No automated backup, you have to do it yoursel...

Singleton Design Principle for pyspark database connector A singleton is a design pattern that ensures that a class has only one instance, and provide...

Singleton Design Principle for pyspark database connectorA singleton is a design pattern that ensures that a class has only one instance, and provides a global access point to that instance. Here is an example of how you could implement a singleton d...

How to use partial_parse.msgpack with workflow dbt task?

I'm looking for direction on how to get the dbt task in workflows to use the partial_parse.msgpack file to skip parsing files that haven't changed. I'm downloading my artifacts after each run and the partial_parse file is being saved back to adls.Wha...

Read/Write concurrency issue

Hi. I assume that it can be concurrency issue. (a Read thread from Databricks and a Write thread from another system)From the start:I read 12-16 csv files (approximately 250Mb each of them) to dataframe. df = spark.read.option("header", "False").opti...

Is there a way to automate the Azure Databricks dashboard to schedule and send an email output with the dashboard?

Spark SQL & Spark ML

I am using Spark SQL to import their data into a machine learning pipeline. Once data is imported I want performs machine learning tasks using Spark ML. So I should use what compute tools is best suited for this use case? Please help me!!! Thank you ...

I am using Spark SQL to import their data into a machine learning pipeline. Once data is imported I want performs machine learning tasks using Spark...

I am using Spark SQL to import their data into a machine learning pipeline. Once data is imported I want performs machine learning tasks using Spark ML. So I should use what compute tools is best suited for this use case? Please help me!!! Thank y...

How to Enable Files in Repos in DBR 7.3 LTS ML ?

we need a custom version of a GPU cluster with following requirements for a certain project:Ubuntu 18.04Cuda 10.1.Tesla T4 GPU.Availability of /Workspace/Repos folder.All of these requirements are available with DBR ML 7.3 LTS. But one critical compo...

How to access image data for yolov5 object detection module

How to access image data for the yolov5 object detection module, I am unable to do so.

How to find the road distance between two cities? We can use Azure databricks and azure map.

We will be given an excel file, in which we can see column sender_city and destination_city. We have to find the distance between these two cities and the distance calculated should be written in a column total_distance. All these processes should be...

Auto Loader: Empty fields (discovery_time, commit_time, archive_time) in cloud_files_state

Hi! Why are the fields discovery_time, commit_time, and archive_time NULL in cloud_files_state? Do I need to configure anything when creating my Auto Loader? df = spark.readStream.format("cloudFiles") \ .option("cloudFiles.format", "json") \ ...